Coupling ICP and Whole Image Alignment for Real-time Camera Tracking

Tracking a camera’s movement through space while compiling a map of the observed environment, commonly called Simultaneous Localization And Mapping (SLAM), is a central task in robotics and computer vision. Many variants of this task exist, depending on the available sensor data, whether it has to run at interactive rates or with offline data, and the desired properties of the resulting scene map. A particularly challenging form of SLAM, in which the only available data source is a video stream from a single-lens RGB camera that has to be tracked live, is called monocular SLAM or realtime Structure from Motion. It combines the difficulties of unknown camera pose, unknown structure of the observed scene, and tight bounds on computation time. While being hard, monocular SLAM is also especially interesting, as RGB cameras are ubiquitously available and allow for virtually limitless applications. This work combines two approaches for accurate tracking in a monocular SLAM setting: First, the physical camera is tracked in realtime using a semi-dense frameto-keyframe scheme. By using keyframe depth maps which are dense only where the corresponding input color image exhibits significant visual structure, this scheme minimizes the time spent on data that does not contribute useful tracking information. Areas displaying little or no structure are ignored. Second, newly constructed depth maps are aligned to the existing collection of depth maps using a variant of ICP that is robust with respect to incomplete overlap between data sets, which helps reduce camera drift left by frame-to-keyframe tracking. The tracked input frames are used in realtime to incrementally estimate dense depth maps for new keyframes, thus extending the model and allowing the camera to explore more parts of the scene.